Skip to content

f32 #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
songololo wants to merge 3 commits into
base: main
Choose a base branch
from
Open

f32 #23

songololo wants to merge 3 commits into from

Conversation

@songololo
Copy link
Collaborator

@songololo songololo commented Oct 21, 2025

#22

Makes use of f32 for reading, writing, internal arrays.

@songololo songololo mentioned this pull request Oct 21, 2025
Open
@biglimp
Copy link
Collaborator

biglimp commented Oct 24, 2025

Hi Gareth, can you please to clarify how these changes affect our code in UMEP for QGIS. As far as we can see, you have changed e.g. solweig_run.py. Have you tested this in QGIS environment? I think maybe we have missed this from a previous PR. Why are you making changes in dailyshading.py as this is not used in SOLWEIG? Please clarify?

@songololo
Copy link
Collaborator Author

songololo commented Oct 27, 2025 •
edited
Loading

Hi @biglimp - for this pull request I tried to change all instances of saving rasters to use f32 by default. I searched for all instances of common.save_raster lines where I added coerce_f64_to_f32=True which is a new setting to cast f64 arrays to f32 when saving. If preferred these can be set to False to leave as f64.

The changes to dailyshading.py were to the common.save_raster lines. When saving the file on my vscode it auto formats using the recommended ruff formatter, so the other changed lines are to do with formatting; e.g. ((vegdem2 * vegdem)) became (vegdem2 * vegdem) and int(0) became 0. It also automatically wraps or unwraps lines according to the formatting spec.

The changes to Solweig_run.py are similar (coerce_f64_to_f32=True).

What would be good is to find a way to test the common.py file using some automatic tests since this file needs to automatically detect whether it is running with access to rasterio or not, in which case we assume it is using gdal from QGIS. I'm not sure how to simulate a QGIS environment for testing but if anyone has ideas it would be nice to get some tests setup so that they automatically run when creating a pull request.

Another nice change would be to separate old code into a separate directory and to then auto format all current code using ruff which will help catch some potential errors like the f-string errors that pop up from time to time.

@biglimp
Copy link
Collaborator

biglimp commented Oct 28, 2025

Hi. Thanks for this info. I think you point towards the issue that we cannot control. There is no good way to test within a QGIS environment without doing it manually.

On another note, @nilswallenberg and I are discussing to create a separate pypi for the solweig model consisting of only the model itself, without any other workflow tools such as SVF-calculator. Then we can all use that pypi module to run the actual model. Do you see any issues with this?

@songololo
Copy link
Collaborator Author

songololo commented Nov 19, 2025

@biglimp apologies for the slow response.

At the moment umep-core is designed to work either from Python or from QGIS (not tested at this point).

For this to work, the common.py module is intended to check whether running in QGIS or not and to execute reading and writing operations accordingly (GDAL vs rasterio).

Personally I would recommend to keep SOLWEIG and other core functions for SVF and shadows together so that it isn't necessary to install and manage multiple packages. Though it would be easier to maintain if old or dead code is removed (or moved to a deprecated folder).

@biglimp
Copy link
Collaborator

biglimp commented Nov 24, 2025

@biglimp apologies for the slow response.

At the moment umep-core is designed to work either from Python or from QGIS (not tested at this point).

For this to work, the common.py module is intended to check whether running in QGIS or not and to execute reading and writing operations accordingly (GDAL vs rasterio).

Personally I would recommend to keep SOLWEIG and other core functions for SVF and shadows together so that it isn't necessary to install and manage multiple packages. Though it would be easier to maintain if old or dead code is removed (or moved to a deprecated folder).

Sorry for late reply. I see your point but nevertheless, SVF and wall algorithm are separate tools that users make use of for many other applications a part from SOLWEIG-calculation. SOLWEIG is a separate model and therefore, it would be better to maintain this separately.

@biglimp
Copy link
Collaborator

biglimp commented Nov 27, 2025

Pulled the f32 branch and tried in the osgeo env. Got the following error:

C:\Users\xlinfr\Documents\PythonScripts\umep-core>C:/OSGeo4W/bin/python.exe c:/Users/xlinfr/Documents/PythonScripts/umep-core/demos/solweig_gbg_test.py
Traceback (most recent call last):
  File "C:\Users\xlinfr\Documents\PythonScripts\umep-core\umep\common.py", line 11, in <module>       
    import rasterio
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\rasterio\__init__.py", line 25, in <module>       
    from rasterio._base import DatasetBase
ImportError: DLL load failed while importing _base: The specified procedure could not be found.       

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\osgeo\__init__.py", line 30, in swig_import_helper
    return importlib.import_module(mname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 921, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 813, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1293, in create_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
ImportError: DLL load failed while importing _gdal: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\xlinfr\Documents\PythonScripts\umep-core\demos\solweig_gbg_test.py", line 4, in <module>
    from umep import (
  File "C:\Users\xlinfr\Documents\PythonScripts\umep-core\umep\skyviewfactor_algorithm.py", line 53, in <module>
    from umep import class_configs, common
  File "C:\Users\xlinfr\Documents\PythonScripts\umep-core\umep\class_configs.py", line 19, in <module>
    from . import common
  File "C:\Users\xlinfr\Documents\PythonScripts\umep-core\umep\common.py", line 21, in <module>
    from osgeo import gdal
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\osgeo\__init__.py", line 35, in <module>
    _gdal = swig_import_helper()
            ^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\Python312\Lib\site-packages\osgeo\__init__.py", line 32, in swig_import_helper
    return importlib.import_module('_gdal')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\OSGeo4W\apps\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_gdal'

C:\Users\xlinfr\Documents\PythonScripts\umep-core>

Here are the python packages installed (python 3.12.9):

C:\Users\xlinfr\Documents\PythonScripts\umep-core>pip list
Package                       Version
----------------------------- -------------
affine                        2.4.0
alabaster                     0.7.16
annotated-types               0.7.0
anyio                         4.3.0
appdirs                       1.4.4
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
asteval                       1.0.0
asttokens                     2.4.1
async-lru                     2.0.4
atmosp                        0.2.9
attrs                         23.2.0
Babel                         2.14.0
beautifulsoup4                4.12.3
bleach                        6.1.0
cads-api-client               1.0.3
cdsapi                        0.7.6
certifi                       2024.2.2
cffi                          1.16.0
cftime                        1.6.4
chardet                       5.2.0
charset-normalizer            3.3.2
click                         8.1.7
click-plugins                 1.1.1
cligj                         0.7.2
cloudpickle                   3.0.0
colorama                      0.4.6
comm                          0.2.2
contourpy                     1.2.1
coverage                      7.4.4
crc32c                        2.7.1
cycler                        0.12.1
dask                          2024.6.2
datapi                        0.1.1
debugpy                       1.8.5
decorator                     5.1.1
defusedxml                    0.7.1
dill                          0.3.8
docutils                      0.20.1
donfig                        0.8.1.post1
ecmwf-datastores-client       0.1.0
et_xmlfile                    1.1.0
executing                     2.0.1
ExifRead                      3.0.0
f90nml                        1.4.4
f90wrap                       0.2.16
fastjsonschema                2.19.1
fiona                         1.10.1
flexcache                     0.3
flexparser                    0.3.1
fonttools                     4.51.0
fsspec                        2024.6.1
future                        1.0.0
GDAL                          3.10.2
geographiclib                 2.0
geopandas                     1.0.1
h11                           0.14.0
h3                            4.3.1
h5netcdf                      1.6.1
h5py                          3.11.0
httpcore                      1.0.5
httplib2                      0.22.0
httpx                         0.27.0
idna                          3.7
imagesize                     1.4.1
importlib_metadata            7.1.0
ipykernel                     6.29.5
ipython                       8.26.0
ipywidgets                    8.1.2
JayDeBeApi                    1.2.3
jedi                          0.19.1
Jinja2                        3.1.3
joblib                        1.4.0
JPype1                        1.5.0
json5                         0.9.25
jsonschema                    4.21.1
jsonschema-specifications     2023.12.1
jupyter                       1.1.1
jupyter_client                8.6.2
jupyter-console               6.6.3
jupyter_core                  5.7.2
jupyter-events                0.10.0
jupyter-lsp                   2.2.5
jupyter_server                2.14.0
jupyter_server_terminals      0.5.3
jupyterlab                    4.1.5
jupyterlab_pygments           0.3.0
jupyterlab_server             2.26.0
jupyterlab_widgets            3.0.10
kiwisolver                    1.4.5
linecache2                    1.0.0
llvmlite                      0.42.0
lmfit                         1.3.1
locket                        1.0.0
lxml                          5.3.0
markdown-it-py                4.0.0
MarkupSafe                    2.1.5
matplotlib                    3.10.0
matplotlib-inline             0.1.7
mdurl                         0.1.2
mistune                       3.0.2
mock                          5.1.0
multiprocess                  0.70.16
multiurl                      0.3.3
munch                         4.0.0
nbclient                      0.10.0
nbconvert                     7.16.3
nbformat                      5.10.4
nest-asyncio                  1.6.0
netCDF4                       1.7.1.post1
networkx                      3.3
nose                          1.3.7
nose2                         0.14.1
notebook                      7.1.2
notebook_shim                 0.2.4
numba                         0.59.0
numcodecs                     0.16.0
numdifftools                  0.9.41
numexpr                       2.10.1
numpy                         1.26.2
odfpy                         1.4.1
oldest-supported-numpy        2023.12.21
openpyxl                      3.1.2
overrides                     7.7.0
OWSLib                        0.32.0
packaging                     24.0
pandas                        2.2.2
pandocfilters                 1.5.1
parso                         0.8.4
partd                         1.4.2
pb-tool                       3.1.0
pillow                        11.1.0
Pint                          0.24.1
pip                           25.2
platformdirs                  4.2.2
Platypus-Opt                  1.0.4
plotly                        5.20.0
ply                           3.11
pooch                         1.8.2
prometheus_client             0.20.0
prompt_toolkit                3.0.47
psutil                        6.0.0
psycopg2                      2.9.10
pure_eval                     0.2.3
pvlib                         0.11.0
pyarrow                       22.0.0
pycparser                     2.22
pydantic                      2.11.7
pydantic_core                 2.33.2
Pygments                      2.17.2
pyodbc                        5.1.0
pyogrio                       0.9.0
PyOpenGL                      3.1.7
pyparsing                     3.1.2
PyPDF2                        3.0.1
pypiwin32                     223
pyproj                        3.7.0
PyQt5                         5.15.11
PyQt5_sip                     12.16.1
PyQtWebEngine                 5.15.7
pyserial                      3.5
python-dateutil               2.9.0.post0
python-json-logger            2.0.7
pytz                          2024.1
pywin32                       306
pywinpty                      2.0.13
PyYAML                        6.0.1
pyzmq                         26.2.0
qtconsole                     5.6.1
QtPy                          2.4.1
rasterio                      1.4.3
referencing                   0.34.0
remotior_sensus               0.4.4
reportlab                     4.2.5
requests                      2.31.0
retrying                      1.3.4
rfc3339_validator             0.1.4
rfc3986_validator             0.1.1
rich                          14.2.0
rioxarray                     0.18.2
rpds-py                       0.18.0
ruff                          0.12.8
scikit-learn                  1.4.2
scipy                         1.13.0
seaborn                       0.13.2
Send2Trash                    1.8.3
setuptools                    75.3.0
shapely                       2.0.6
simplejson                    3.19.2
sip                           6.9.1
six                           1.16.0
sniffio                       1.3.1
snowballstemmer               2.2.0
soupsieve                     2.5
Sphinx                        7.3.7
sphinx-rtd-theme              2.0.0
sphinxcontrib-applehelp       1.0.8
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.0.5
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.7
sphinxcontrib-serializinghtml 1.1.10
stack-data                    0.6.3
supy                          2025.11.20rc1
target-py                     0.1.1
tenacity                      8.2.3
terminado                     0.18.1
threadpoolctl                 3.4.0
timezonefinder                6.5.9
tinycss2                      1.2.1
toml                          0.10.2
toolz                         0.12.1
tornado                       6.4.1
tqdm                          4.66.4
traceback2                    1.4.0
traitlets                     5.14.3
typing_extensions             4.14.0
typing-inspection             0.4.1
tzdata                        2025.2
tzfpy                         1.1.0
uncertainties                 3.2.1
unittest2                     1.1.0
urllib3                       2.2.1
uv                            0.6.8
uwg                           5.8.13
wcwidth                       0.2.13
webencodings                  0.5.1
websocket-client              1.7.0
widgetsnbextension            4.0.10
wxPython                      4.2.2
xarray                        2025.3.0
xlrd                          2.0.1
xlwt                          1.3.0
zarr                          3.0.8
zipp                          3.18.1

Am I missing something?

@biglimp
Copy link
Collaborator

biglimp commented Nov 28, 2025

Just made a simple test so that it was not my VSCode who did something strange. the code below:

import sys, os, rasterio
print("PYTHON:", sys.executable)
print("RASTERIO:", os.path.dirname(rasterio.__file__))
print("PATH:", os.environ["PATH"].split(";"))

gives below with no errors:

C:\Users\xlinfr\Documents\PythonScripts\umep-core>C:/OSGeo4W/bin/python.exe c:/Users/xlinfr/Documents/PythonScripts/umep-core/umep/importtest.py
PYTHON: C:\OSGeo4W\bin\python.exe
RASTERIO: C:\OSGeo4W\apps\Python312\Lib\site-packages\rasterio
PATH: ['C:\\OSGeo4W\\apps\\qt5\\bin', 'C:\\OSGeo4W\\apps\\Python312\\Scripts', 'C:\\OSGeo4W\\bin', 'C:\\windows\\system32', 'C:\\windows', 'C:\\windows\\system32\\WBem', 'C:\\OSGeo4W\\apps\\qgis\\bin', 'C:\\OSGeo4W\\apps\\Qt5\\bin', 'C:\\OSGeo4W\\apps\\Python312\\Scripts', 'c:\\Users\\xlinfr\\.vscode\\extensions\\ms-python.debugpy-2025.16.0-win32-x64\\bundled\\scripts\\noConfigScripts']

@songololo
Copy link
Collaborator Author

songololo commented Jan 16, 2026 •
edited
Loading

Thanks for the detailed report and for testing this. [EDIT - updated below in the later response.]

The suspected issue

OSGeo4W includes its own GDAL installation. When rasterio is pip-installed into that environment, it brings bundled GDAL DLLs that conflict with OSGeo4W's version. It might have been that the standalone rasterio test worked because it loaded before the conflict was triggered, but umep's fallback logic exposes the incompatibility.

Solution

Ideally, use a separate virtual environment instead of the OSGeo4W Python:

cd C:\Users\<path>\Documents\PythonScripts\umep-core

uv venv --python 3.12
.venv\Scripts\activate
uv pip install -e .

python demos/solweig_gbg_test.py

Or with standard venv:

python -m venv .venv
.venv\Scripts\activate
pip install -e .

Alternative: Conda

conda-forge maintains compatible versions of the geospatial stack:

conda create -n umep -c conda-forge python=3.12 rasterio geopandas pyproj shapely
conda activate umep
pip install -e .

Diagnostic tool

In the more recent tiler branch I've added umep-doctor to help identify environment issues. After pulling the latest:

umep-doctor

Let me know if you run into anything else.

@songololo
Copy link
Collaborator Author

songololo commented Jan 16, 2026

Sorry for late reply. I see your point but nevertheless, SVF and wall algorithm are separate tools that users make use of for many other applications a part from SOLWEIG-calculation. SOLWEIG is a separate model and therefore, it would be better to maintain this separately.

Hi @biglimp - Happy New Year. Sorry I've been quiet.

Shall I go ahead and create a new separate repository to split out the Solweig logic?

This would make it easier to maintain if we can do a fresh, clean start.

I would ideally also like to simplify the function calls with a new logic, but would like to also keep it compatible with any other wrappers, e.g. QGIS that might rely on it. Please let me know if you have any thoughts.

@songololo
Copy link
Collaborator Author

songololo commented Jan 17, 2026 •
edited
Loading

Pulled the f32 branch and tried in the osgeo env. Got the following error:

Hi @biglimp - I took another look at this assuming you'd like to run it directly in the OSGEO4W environment: The common.py file in the tiler branch now has additional logic which will try to make an OSGEO4W environment work even if there are some conflicting package installs like rasterio (which has conflicting dlls).

Could you please give this a go and let me know if it works for you? (The latest changes are in the tiler branch which is ahead of the f32 branch.)

@biglimp
Copy link
Collaborator

biglimp commented Jan 19, 2026

Sorry for late reply. I see your point but nevertheless, SVF and wall algorithm are separate tools that users make use of for many other applications a part from SOLWEIG-calculation. SOLWEIG is a separate model and therefore, it would be better to maintain this separately.

Hi @biglimp - Happy New Year. Sorry I've been quiet.

Shall I go ahead and create a new separate repository to split out the Solweig logic?

This would make it easier to maintain if we can do a fresh, clean start.

I would ideally also like to simplify the function calls with a new logic, but would like to also keep it compatible with any other wrappers, e.g. QGIS that might rely on it. Please let me know if you have any thoughts.

Hi. Yes, that would be good. This will make it much easier to incorporate changes in the model, separate from e.g. Walls algorithms etc. @nilswallenberg , and me are also in contact with a group in the US who have made a GPU version of the code that we would like to include. Nils, could you add the link to the repo that has this code or is it non-public still?

@biglimp
Copy link
Collaborator

biglimp commented Jan 19, 2026

Pulled the f32 branch and tried in the osgeo env. Got the following error:

Hi @biglimp - I took another look at this assuming you'd like to run it directly in the OSGEO4W environment: The common.py file in the tiler branch now has additional logic which will try to make an OSGEO4W environment work even if there are some conflicting package installs like rasterio (which has conflicting dlls).

Could you please give this a go and let me know if it works for you? (The latest changes are in the tiler branch which is ahead of the f32 branch.)

Thanks for this. I will try this as soon as I find the time. @nilswallenberg , if you have time to try this before me, please let me know the outcome.

@nilswallenberg
Copy link

nilswallenberg commented Jan 20, 2026

Hi,

This is their repo: https://github.com/nvnsudharsan/SOLWEIG-GPU
and it's available as a Python package: https://pypi.org/project/solweig-gpu/

Cheers,
Nils

@nilswallenberg
Copy link

nilswallenberg commented Jan 28, 2026

Hi @songololo

I tried running in the OSGEO4W Python environment with success. I only ran into problems with "from pvlib.iotools import read_epw", but running without it worked without issues.

QGIS version: 3.44.3-Solothurn
Qt version: 5.15.13
Python version: 3.12.11
GDAL version: 3.11.3 — Eganville
OS version: Windows 11 Version 2009

Cheers,
Nils

@biglimp
Copy link
Collaborator

biglimp commented Jan 28, 2026

Hi @songololo

I tried running in the OSGEO4W Python environment with success. I only ran into problems with "from pvlib.iotools import read_epw", but running without it worked without issues.

QGIS version: 3.44.3-Solothurn Qt version: 5.15.13 Python version: 3.12.11 GDAL version: 3.11.3 — Eganville OS version: Windows 11 Version 2009

Cheers, Nils

Many thanks. If we fix that import we could finally merge this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@songololo @biglimp @nilswallenberg